Ran the siderophile reproduction + re-enabled smoke test

The PR note said cargo wasn't installed here so the end-to-end siderophile reproduction and the re-enabled smoke test (#181) weren't run. I ran them with the rust toolchain (cargo 1.96.1) installed, applying this PR plus #181's re-enabled smoke test on top of master.

Reproduction — confirms the fix works

pytest test/test_smoke.py::TestSmoke::test_cargo --runintegration (10m48s, resolves trailofbits/siderophile@7bca0f5):

• Resolution expands to 971 packages, up from 129 on current master / 163 in the committed baseline. This matches the ~963 figure in Cargo resolver prunes optional/feature-gated dependencies (collapses "all possible versions" to the locked tree) #189 and verifies the fix restores "resolve all possible versions" for cargo.
• The new CalledProcessError handling fired as designed: 2 cargo metadata calls failed on fully-yanked transitive deps (cloudflare-zlib-sys ^0.2.0, core2 ^0.4) and were logged + skipped instead of aborting the whole run (the pre-fix behavior that Add exception handling #125's pattern addresses).
• test/test_cargo.py: 7 passed, including the new test_resolve_from_source_only_marks_resolved_deps.

Smoke test — fails on one invariant, but it's baseline drift, not a regression

The only failure is:

AssertionError: resolution missing expected package(s) ['cargo:cc']

Root cause, verified from the resolved graph:

• In the 163-package baseline, cc was reachable only via afl@0.10.1 → cc.
• The full expansion now selects newer afl (0.15.24, 0.16.0), which dropped the cc build-dependency (afl now deps libc, rustc_version, xdg).
• The remaining paths to cc (flate2 → cloudflare-zlib-sys, bitstream-io → core2) hit the fully-yanked crates above, so those subtrees legitimately error out.

So cc genuinely isn't in the current resolution. The baseline siderophile.expected.json predates both this fix and the current crates.io state — this is exactly the drift the "open question" here and #181 anticipate. The gap needs the semantics decision plus a re-baseline via test/rebuild_expected_output.py, not a code change. I did not rebuild the baseline, since that's the decision this draft defers.

🤖 Generated with Claude Code

I am a bit conflicted on the optional dependency semantics decision. My current instinct is to ignore optional dependencies that aren't explicitly required. It's entirely possible that a package could have an incompatible subset of optional dependencies; so if we force them all to be expanded, there might be no valid resolution! In other words, there might be a mutually exclusive set of optional dependencies. @evandowning What do you think? Our decision on this will likely affect the order in which we merge this PR versus #181.

@ESultanik That's a great point. I'll change this to ignore optional dependencies.

Pushed a commit implementing the "ignore optional dependencies that aren't explicitly required" semantics from the review.

What changed

get_dependencies now skips any dependency with "optional": true in cargo metadata, alongside the existing dev/build (kind) filter. Optional deps never become edges in the graph, so they are neither expanded nor pruned. Only deps cargo requires under the active feature set are tracked.

This directly addresses the mutual-exclusivity concern: a crate can declare a mutually exclusive set of optional deps (e.g. sqlx's runtime-tokio-rustls vs runtime-async-std-native-tls backends, guarded by a compile-time check), and forcing them all into one resolution would describe a configuration that can never build. Ignoring inactive optional deps avoids ever constructing such a set.

The resolve_from_source match guard is kept but its purpose narrows: with optional deps filtered upstream, it now only protects a required dep that cargo failed to resolve (e.g. a subcrate whose cargo metadata errored) — leaving it for normal resolution rather than marking it resolved-to-nothing and pruning it. The CalledProcessError handling from the first commit is unchanged and still needed.

Tests

• Added test_get_dependencies_ignores_optional_deps: mocks cargo metadata output and asserts an optional normal dep (rayon) and a dev dep (criterion) are dropped while a required dep (serde) survives.
• Reframed test_resolve_from_source_only_marks_resolved_deps to reflect the narrowed guard (required-but-unresolved deps, not optional deps).

test/test_cargo.py: 8 passed. ruff + mypy clean.

Tradeoff worth flagging

This is the conservative end of the semantics spectrum. Because cargo marks a dep optional: true even when a default feature enables it, an optional dep reachable only via optional edges is now dropped — including the ~34 crates (rayon, backtrace, afl, …) that #189 noted were missing. So the resolved set returns to roughly the locked-tree size rather than the ~971-package full expansion. That is the intended effect of this semantics choice, but it does mean the re-baselined siderophile.expected.json (and the #181 smoke test) should be regenerated against this behavior, not the full-expansion behavior.